1
00:00:00,540 --> 00:00:01,070
All right.

2
00:00:01,080 --> 00:00:07,560
So in this lesson we're going to start talking about how to evaluate our model.

3
00:00:07,860 --> 00:00:13,650
We've built that naive based classifier from scratch and we've trained it and classified our emails

4
00:00:14,070 --> 00:00:21,470
in our testing dataset but we don't know yet if our model is doing a good job or a bad job.

5
00:00:21,480 --> 00:00:26,880
Speaking of jobs remember that the premise of this whole project is that we're applying for a job at

6
00:00:26,880 --> 00:00:32,670
our favorite company and they've asked all the candidates to complete a machine learning exercise as

7
00:00:32,670 --> 00:00:36,580
part of the application process and to land that job.

8
00:00:36,630 --> 00:00:39,070
We want to be able to show off a good model right.

9
00:00:39,870 --> 00:00:41,640
But here's the catch.

10
00:00:41,640 --> 00:00:44,880
What does good actually mean.

11
00:00:44,880 --> 00:00:50,390
Well if an email was spam then our model should classified a spam right.

12
00:00:51,160 --> 00:00:57,980
And if an email was legitimate then our model should classify this email as non spam.

13
00:00:58,170 --> 00:01:01,140
That's not too much to ask right.

14
00:01:01,190 --> 00:01:08,340
Well I think the first thing to check is how many emails were actually classified correctly.

15
00:01:08,340 --> 00:01:15,360
What is the fraction of the predictions that our model got right in this lesson we'll be looking at

16
00:01:15,360 --> 00:01:23,810
the models accuracy let's add a section heading here that reads Metrics and Evaluation

17
00:01:26,300 --> 00:01:30,190
and as a subheading I'll add accuracy.

18
00:01:30,530 --> 00:01:33,890
This will be the first metric that we're going to look at.

19
00:01:33,890 --> 00:01:41,510
So how many emails that our model classify correctly and how many emails that our model classify incorrectly.

20
00:01:41,510 --> 00:01:45,290
Well the number of correct documents correct.

21
00:01:45,290 --> 00:01:48,240
And it's called docs will be equal to.

22
00:01:48,290 --> 00:01:50,690
Well I'll be equal to where y underscore.

23
00:01:50,690 --> 00:01:55,880
Test is equal to our prediction.

24
00:01:55,910 --> 00:01:58,640
Dot some.

25
00:01:58,670 --> 00:02:05,780
In other words we're taking the true values and we're comparing them with the double equal sign whether

26
00:02:05,780 --> 00:02:11,070
they're equal to our vector of predictions.

27
00:02:11,120 --> 00:02:14,250
This is what we've calculated up here.

28
00:02:14,720 --> 00:02:21,560
And because the result of this comparison will either be true or false summing up all the values which

29
00:02:21,650 --> 00:02:25,160
are true will be the number of correct documents.

30
00:02:25,160 --> 00:02:26,480
So let's print this out.

31
00:02:26,630 --> 00:02:42,340
Print docs classified correctly comma correct docs and yes that's a better variable name.

32
00:02:42,750 --> 00:02:43,970
And let's hit shift into.

33
00:02:45,540 --> 00:02:53,510
So we've got one thousand six hundred and seventy one documents that were classified correctly now suppose

34
00:02:53,510 --> 00:02:58,310
we want to know the number of documents that were classified incorrectly.

35
00:02:58,310 --> 00:03:05,470
I can store this in a variable called say num docs on the score wrong and that would be equal to.

36
00:03:06,350 --> 00:03:11,870
Well it'll be the number of rows the number of entries and X underscore test which we can get with X

37
00:03:11,880 --> 00:03:24,060
and court test dot shape square brackets 0 minus the number of correct documents and that if we printed

38
00:03:24,100 --> 00:03:34,330
out see Doc's classified incorrectly is equal to 52.

39
00:03:34,580 --> 00:03:36,730
Now that seems pretty promising right.

40
00:03:36,760 --> 00:03:42,340
Because the interesting thing about these results is that our model is classifying e-mails that it has

41
00:03:42,340 --> 00:03:43,690
never seen before.

42
00:03:44,110 --> 00:03:50,380
We've trained our model on a completely different data set of e-mails and still out of over a thousand

43
00:03:50,380 --> 00:03:51,660
seven hundred e-mails.

44
00:03:51,730 --> 00:03:55,050
It only misclassified 52 of them.

45
00:03:55,810 --> 00:04:03,310
So now that we've calculated the number of e-mails classified correctly and incorrectly how do we quantify

46
00:04:03,610 --> 00:04:06,720
the model's accuracy.

47
00:04:06,750 --> 00:04:12,050
Well let's look at the definition of the accuracy metric in machine learning.

48
00:04:12,350 --> 00:04:21,500
The mean accuracy on the test data is defined as the number of documents classified correctly over the

49
00:04:21,500 --> 00:04:24,210
total number of documents.

50
00:04:24,230 --> 00:04:30,950
So we've got our number of correct predictions divided by the total number of predictions.

51
00:04:31,250 --> 00:04:35,060
That's what accuracy means in machine learning.

52
00:04:35,060 --> 00:04:37,250
Let's calculate that here for good measure.

53
00:04:37,250 --> 00:04:46,820
So a lot a little common here that reads accuracy and it's gonna be the correct documents divided by

54
00:04:47,000 --> 00:04:49,330
the total number of documents.

55
00:04:49,370 --> 00:04:54,230
In other words the length of X on a school test.

56
00:04:54,230 --> 00:05:02,370
This will give us the same answer as this the number of rows in our testing dataset let's see what it

57
00:05:02,370 --> 00:05:05,470
is I had around 97 percent.

58
00:05:05,880 --> 00:05:07,160
It's actually not all that bad.

59
00:05:07,160 --> 00:05:13,590
Right now no other way to calculate the accuracy is to look at the fraction of e-mails that were classified

60
00:05:13,680 --> 00:05:15,200
incorrectly.

61
00:05:15,240 --> 00:05:26,070
So if I say fraction of a school wrong is equal to num docs wrong divided by alien X on a school test.

62
00:05:26,070 --> 00:05:28,960
That's the fraction of documents that we've got wrong.

63
00:05:29,260 --> 00:05:41,550
And if we print this out say print parentheses fraction classified incorrectly is and I want to provide

64
00:05:41,550 --> 00:05:43,950
this percentage to two decimal places.

65
00:05:44,100 --> 00:05:50,820
I can insert some curly braces here and put a colon in there and then dot and then two and then the

66
00:05:50,820 --> 00:05:59,670
percent sign and then after that single quote for the string I can put a dot then format parentheses

67
00:06:00,450 --> 00:06:02,760
fraction underscore wrong.

68
00:06:03,030 --> 00:06:12,430
So this will print out three point zero two percent and that means if we print out the accuracy of the

69
00:06:12,430 --> 00:06:26,250
model is curly braces colon point two percent dot format one minus fraction.

70
00:06:26,250 --> 00:06:27,600
Wrong.

71
00:06:27,600 --> 00:06:28,560
There we go.

72
00:06:28,560 --> 00:06:32,670
Ninety six point nine eight percent.

73
00:06:32,910 --> 00:06:36,510
Now looking at these numbers is very well and good.

74
00:06:36,840 --> 00:06:41,760
And we're going to be looking at a few more metrics as part of our evaluation process.

75
00:06:41,910 --> 00:06:46,550
But you know what I think we should visualize how our model is doing exactly.

76
00:06:46,560 --> 00:06:52,930
I think we should plotted and create some sort of chart and we're going to do that in the next lesson.

77
00:06:52,980 --> 00:06:53,790
I'll see you there.